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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 

WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 
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Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )E3 Responsive to communication(s) filed on 12 January 2007 . 
2a)^ This action is FINAL. 2b)D This action is non-final. 

3) Q Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 
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5) D Claim(s) is/are allowed. 

6) (3 Claim(s) 1,3-15,19.20.24 and 31-34 is/are rejected. 

7) £<] Claim(s) 21.22 and 25 is/are objected to. 

8) Q Claim(s) are subject to restriction and/or election requirement. 

Application Papers 
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Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
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DETAILED ACTION 

1 . This action is responsive to communications: amendment filed 1/12/2007, to the original application 
filed 2/6/2002, drawings filed 3/25/2002. The present application is a continuation (filed under 37 CFR 1.53(b)) 
of parent case 08/900,421 (now abandoned). Said parent case filed 7/25/1997. 

2. Claims 1, 3-15, 19-22, 24, 25, 31-34 pending. Claims 1, 14, 19, 20 are independent claims. 

Allowable Subject Matter 

3. Claims 21, 22, and 25 are objected to as being dependent upon a rejected base claim, but would be 
allowable if rewritten in independent form including all of the limitations of the base claim and any intervening 
claims. 

Claim Rejections - 35 USC§ 103 

4. The following is a quotation of 35 U.S.C 103(a) which forms the basis for all obviousness 

rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the 
differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall 
not be negatived by the manner in which the invention was made. 

5. Claims 1, 6-13, 16-17, 19, 23-24, 26, 28, 29 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Mendelson et al. (hereinafter Mendelson), U.S. Patent No. 5,754,783 issued May 1998, 
in view of Durward et al. (hereinafter Durward), U.S. Patent No. 5,950,202 issued September 1999, and in 
view of Nielsen (hereinafter Nielsen), U.S. Patent No. 5,805,153 issued September 1998. 
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In regard to independent claims 1, and 26, Mendelson teaches interleaving (i.e. synchronizing) timed 
primary and untimed secondary data streams at a server, said resulting stream(s) eventually transported to clients 
for display (Mendelson Abstract, column 4 lines 18-22, column 8 lines 15-20). Mendelson defines timed 
primary content data as games, movies, audio music, etc., acquired and stored in a storage device (hard drive) on 
said server (Mendelson column 1 lines 13-22, column 6 lines 8-1 1, Figure 4 item 421). Mendelson defines 
untimed secondary content as video text, graphic images, private data such as files, databases, or stuffing bits, 
etc. (Mendelson column 1 lines 24-28, column 3 lines 50-54, column 6 lines 23-26, Figure 4 item 414). It is 
noted that any number of data streams can be utilized (Mendelson column 4 lines 54-58). 

Mendelson does not specifically teach motion "capture" data. However, Durward teaches a virtual 
reality database for coordinating a three-dimensional virtual space (Durward Abstract). Since communication of 
changed reference motion data is essentially real-time (Durward Figures 1, 4), said motion data stream is 
captured accordingly. It would have been obvious to one of ordinary skill in the art at the time of the invention 
to apply Durward to Mendelson, providing the benefit of motion data capture for a realistic virtual reality 
experience. 

Mendelson does not specifically disclose "motion" type data. However, Mendelson's secondary content 
(video text/images etc.) can at least be interpreted as suggestive of motion data since it is displayed as a 
video/movie type stream. In addition, Mendelson's secondary content can also comprise "padding", null/stuffing 
data, interleaved via a stuffing buffer, helping to preserve the quality (i.e. motion - wander, jitter, etc.) of the 
timed primary data (Mendelson column 2 lines 50-67, column 3 lines 9-20, 50-54, column 8 lines 14-17, Figure 
4 item 440). It would have been obvious to one of ordinary skill in the art at the time of the invention to interpret 
Mendelson's secondary data as a type of "motion" data, providing the benefit of controlling/minimizing possible 
timed motion related problems (i.e. wandering, jittering, etc.) of the primary content (compare the above two 
paragraphs with claim 1 "A method of synchronizing asynchronous time-based and motion data in a system in 
which the time-based data and the motion data are transmitted by a server over a network to a client, the 
method comprising", and "retrieving a time-based data stream and a motion data stream at the server..") 
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Mendelson teaches receiving various time-based and motion data at a server, as explained above. In 
addition, since Mendelson discloses typical timed and untimed content within a frames environment (Mendelson 
column 1 lines 17-18, 24-25), combined with Mendelson' s teaching of using the MPEG standard (typically 
applied to MPEG files, which are frame-based) (Mendelson column 4 lines 48-52), Mendelson's data streams 
can be interpreted as frame based (compare with claim 1 "... each stream comprising frames of data"). 

Mendelson teaches variable buffering (at a server) by measuring the bit rate stream, and buffers via the 
use of variable usage (interleaving) of primary, secondary and stuffing content buffers (Mendelson column 6 
lines 62-67, column 8 lines 6-20, Figure 4 items 440, 450, 460). It is noted that each type of content has its own 
buffer, and are regulated accordingly. It is noted that Mendelson's resulting data stream can be interpreted as 
comprising at least two interleaved streams (now custom synchronized) of various data types. In addition, 
multiple elementary streams within a transport stream can be utilized (Mendelson column 4 lines 54-63, column 
6 lines 4-8) (compare with claim 1 "variable buffering one of the time-based data stream and the motion data 
stream at the server to produce two streams having synchronized frames"). 

Mendelson teaches transport of a final (custom synchronized) stream(s) to customers (clients) for 
eventual playback (Mendelson column 6 lines 4-8, column 8 lines 10-12; compare with claim 1 "using the 
synchronized frames at the client for playback of synchronized motion and time-based data to a user"). 

Mendelson does not specifically teach receiving separately the two streams at the client. However, 
Nielsen teaches receiving at the client two separate streams of data (video stream, and a subtitle stream), which 
are processed accordingly (Nielsen Abstract, column 1 lines 37-57, column 2 lines 52-59). It would have been 
obvious to one of ordinary skill in the art at the time of the invention to apply Nielsen's receiving of separate 
streams to Mendelson's time data and motion data (or in the alternative, Mendelson's time data and Durward's 
motion data), providing Mendelson/Durward the benefit of receiving separate streams so as to facilitate 
adjustment of motion data independently from the timed data (see Nielsen Abstract - at bottom, also column 5 
lines 55-61). 
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In regard to dependent claim 6, Mendelson teaches secondary data as stuffing data, or as private data 
such as files or databases (see rejection of claim 1, also Mendelson column 6 lines 24-26). Mendelson does not 
specifically teach said motion data mapped to control movement of a virtual figure displayed in a scene at the 
client. However, Durward teaches a method whereby updated positional data from a person's head position 
sensor is mapped and used to determine the position of a virtual being defined for that user, communicating 
graphical data to the user via database data (Durward, column 6 lines 29-32, 46-52, column 7 lines 12-20). It 
would have been obvious to one of ordinary skill in the art at the time of the invention to apply the virtual 
position database data of Durward to Mendelson's secondary database data, providing Mendelson increased 
spatial imagery to the multimedia stream. 

In regard to dependent claim 7, Mendelson does not specifically teach using a body suit for 
generating motion data. However, Durward teaches a method of sensing position data with the use of a body suit 
(Durward, column 3 line 25-33). It would have been obvious to one of ordinary skill in the art at the time of the 
invention to apply Durward to Mendelson, providing Mendelson the benefit of three dimensional physical 
contour mapping, providing increased spatial image accuracy to Mendelson's video stream method. 

In regard to dependent claim 8, Mendelson teaches primary content (video, audio data, etc.) as well as 
secondary content (video text, images, files, database data, along with "stuffing" data, etc.) as explained in the 
rejection of claim 1. As the resulting interleaved stream(s) is played back on a client, any combination of 
foregrounc^ackground combination can be typically specified, (i.e. the primary data (i.e. a movie scene) can be 
run as background to the secondary/motion data controlling it, or run as background to any related database files 
in the foreground, etc). 

In regard to dependent claims 9, 10, Mendelson teaches buffering/processing occurring at the same 
time the program is being transported to customer equipment (Mendelson column 8 lines 9-12; compare with 
claim 9). 
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Mendelson teaches audio (i.e. voice) data (Mendelson column 4 lines 57-59; compare with claim 10). 

In regard to dependent claim 11, Mendelson teaches an embodiment using streams associated with 
data channels (Mendelson column 6 lines l-8).Mendelson also teaches a header packet (a descriptor packet), 
said header typically used to describe data within transport stream data (i.e. each 188 byte packet) (Mendelson 
column 4 lines 47-53, Figure 2). 

Mendelson does not specifically teach that a client may join in progress a multicasting of a stream. 
However, Durward teaches virtual reality presentations where a number of contributing user's data is collected 
and analyzed accordingly, so a to facilitate multicasting (and interaction between figures) of the virtual world to 
various users (Durward Figures 4-6, 7 column 8 lines 5-15). Durward also teaches users logging on to a system 
via modems, said users data is shared with other users and continuously updated so as to create a virtual reality 
world (Durwald, column 8 lines 22-24, 41-64, Figure 1). It would have been obvious to one of ordinary skill in 
the art at the time of the invention to apply Durward to Mendelson, providing Mendelson the benefit of 
multicasting associated with synchronization of streams. 

In regard to dependent claims 12, 13, Mendelson teaches buffering of streams comprising buffering of 
primary and secondary content (said primary content can be audio data, Mendelson column 4 lines 57-59), along 
with necessary interleaved motion type data (secondary/stuffing data), producing a sequenced synchronized 
stream regarding constant verses variable bit stream rate (see also rejection of claim 1, also Mendelson column 6 
lines 62-67). 

In regard to independent claims 16 and 28, claims 16 and 28 reflect the apparatus used to perform the 
methods comprising computer readable instructions as claimed in claim 1, and in further view of the following, 
is rejected along the same rationale. 
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Mendelson does not specifically teach motion "three dimensional" data stream. However, Durward 
teaches a virtual reality database for coordinating a three-dimensional virtual space (Durward Abstract). Since 
communication of changed reference motion data is essentially real-time (Durward Figures 1, 4), said motion 
data stream is captured accordingly and is associated with 3D data. It would have been obvious to one of 
ordinary skill in the art at the time of the invention to apply Durward to Mendelson, providing the benefit of 
motion data 3D capture for a realistic virtual reality experience. 

In regard to dependent claim 17, Mendelson does not specifically teach an apparatus whereby 
synchronized motion/time-based data is milticasted to clients in a network. However, Durward teaches an 
apparatus whereby users log on to a system via modems, said users data is shared with other users and 
continuously updated so as to create a virtual reality world (Durward, column 8 lines 22-24, 41-64, Figure 1). It 
would have been obvious to one of ordinary skill in the art at the time of the invention to apply the data sharing 
method of Durward to Mendelson, providing Mendelson the benefit of continuous updating and increased 
uniform user awareness in a virtual environment. 

In regard to independent claims 19 and 29, claims 19 and 29 incorporate substantially similar subject 
matter as claimed in claim 1, and in further view of the following, is rejected along the same rationale. 

Mendelson does not specifically teach motion "capture" data. However, Durward teaches a virtual 
reality database for coordinating a three-dimensional virtual space (Durward Abstract). Since communication of 
changed reference motion data is essentially real-time (Durward Figures 1, 4), said motion data stream is 
captured accordingly. It would have been obvious to one of ordinary skill in the art at the time of the invention 
to apply Durward to Mendelson, providing the benefit of motion data capture for a realistic virtual reality 
experience. 
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Mendelson teaches secondary data as stuffing data, or as private data such as files or databases (see 
rejection of claim 1, also Mendelson column 6 lines 24-26). Mendelson does not specifically teach said motion 
data mapped to control movement of a virtual figure displayed in a scene at the client, along with 
synchronization. However, Durward teaches a method whereby updated positional data from a person's head 
position sensor is mapped and used to determine the position of a virtual being defined for that user, 
communicating graphical data to the user via database data (Durward, column 6 lines 29-32, 46-52, column 7 
lines 12-20). Since the playback of Mendelson's resulting stream synchronizes the secondary data with the 
primary content, the primary data (i.e. video) can be synchronized with added secondary database data (related 
to a virtual figure). It would have been obvious to one of ordinary skill in the art at the time of the invention to 
apply the virtual position database data of Durward to Mendelson 5 s secondary database data, providing 
Mendelson increased spatial imagery and desired synchronization to the resulting multimedia stream. 

In regard to dependent claim 23, Mendelson does not specifically teach motion "sensor" data. 
However, Durward teaches a virtual reality database for coordinating a three-dimensional virtual space 
(Durward Abstract). Since communication of changed reference motion data is essentially real-time via sensor 
helmets (Durward Figures 1, 4), said motion data stream is captured via sensors accordingly. It would have been 
obvious to one of ordinary skill in the art at the time of the invention to apply Durward to Mendelson, providing 
the benefit of motion data capture for a realistic virtual reality experience. 

In regard to dependent claim 24, Mendelson does not specifically teach motion "sensor" data. 
However, Durward teaches a virtual reality database for coordinating a three-dimensional virtual space 
(Durward Abstract). Since communication of changed reference motion data is essentially real-time via sensor 
helmets (Durward Figures 1, 4), said motion data stream is captured via sensors accordingly. It would have been 
obvious to one of ordinary skill in the art at the time of the invention to apply Durward to Mendelson, providing 
the benefit of motion data capture for a realistic virtual reality experience. 
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6. Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable Mendelson, Dunvard, and 
Nielsen as applied to claim 1 above, and further in view of Baumgartner et al. (hereinafter Baumgartner), 
U.S. Patent No. 5,642,171 issued June 1997. 

In regard to dependent claim 3, Mendelson teaches buffering based on a calculated difference (delay) 
between a variable bit stream, and a constant bit stream (Mendelson column 6 lines 62-67). Mendelson does not 
specifically teach a difference between the delay for motion and time-based data stream to determine buffering 
for a faster stream. However, Baumgartner teaches a method whereby a current video frame number is 
subtracted from a current audio frame number to determine if the audio is too far ahead of the video 
(Baumgartner column 13 lines 60-67, column 14 lines 1-2). It would have been obvious to one of ordinary skill 
in the art at the time of the invention to apply the difference calculation method of Baumgartner to Mendelson's 
stream analysis, providing Mendelson the benefit of more accurate time synchronization to the server buffer 
method disclosed by Mendelson. 

7. Claims 4, 14-15, 18, 27 are rejected under 35 U.S.C. 103(a) as being unpatentable over Mendelson, 
Durward, and Nielsen, and further in view of Shibata, Y., Media synchronization protocols for packet 
audio-video system on multimedia information networks (hereinafter Shibata), IEEE; January 3-6, 1995 
pp.594-601. 

In regard to dependent claim 4, Mendelson does not specifically teach transfer of data values for a 
frame occuring only for a frame that has changed since a last frame was transmitted. However, Shibata discloses 
a method whereby audio data is sent from the video server to the client station only during a "talk spurt", with 
constant frame rates occurring during periods of audio silence (Shibata p.597, section 4.3 "Silence detected 
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synchronization"). It would have been obvious to one of ordinary skill in the art at the time of the invention to 
apply Shibata to Mendelson's stream interleaving, providing Mendelson the benefit of file size and transmission 
reduction (streamlining) of the audio/video data streams. 

In regard to independent claims 14 and 27, claims 14 and 27 incorporate substantially similar subject 
matter as claimed in claim 1, and in further view of the following, is rejected along the same rationale. 

Mendelson does not specifically teach motion "three dimensional" data stream. However, Durward 
teaches a virtual reality database for coordinating a three-dimensional virtual space (Durward Abstract). Since 
communication of changed reference motion data is essentially real-time (Durward Figures 1, 4), said motion 
data stream is captured accordingly and is associated with 3D data. It would have been obvious to one of 
ordinary skill in the art at the time of the invention to apply Durward to Mendelson, providing the benefit of 
motion data 3D capture for a realistic virtual reality experience. 

Mendelson teaches an embodiment using streams associated with "data channels" (Mendelson column 
6 lines l-8).Mendelson also teaches a header packet (a descriptor packet), said header typically used to describe 
data within transport stream data (i.e. each 188 byte packet) (Mendelson column 4 lines 47-53, Figure 2). 

Mendelson does not specifically teach transfer of data values (channels) for a occuring only for changed 
data values. However, Shibata discloses a method whereby audio data is sent from the video server to the client 
station only during a "talk spurt", with constant frame rates occurring during periods of audio silence (i.e. a talk 
spurt changes the data values) (Shibata p.597, section 4.3 "Silence detected synchronization"). It would have 
been obvious to one of ordinary skill in the art at the time of the invention to apply Shibata to Mendelson's 
stream interleaving, providing Mendelson the benefit of file size and transmission reduction (streamlining) of 
the audio/video data streams. It is to be noted that in order to adjust frame rates as above, the server must know 
that a change in the data has occurred, which is typically achieved by comparing new data (the talk spurt) with 
the last recognized data value (previously recognized as empty - reflecting a period of silence). 
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In regard to dependent claim 15, Mendelson teaches an embodiment using streams associated with 
data channels (Mendelson column 6 lines l-8).Mendelson also teaches a header packet (a descriptor packet), 
said header typically used to describe data within transport stream data (i.e. each 188 byte packet) (Mendelson 
column 4 lines 47-53, Figure 2). 

In regard to dependent claim 18, Mendelson teaches an embodiment using streams associated with 
"data channels" (Mendelson column 6 lines l-8).Mendelson also teaches a header packet (a descriptor packet), 
said header typically used to describe data within transport stream data (i.e. each 188 byte packet) (Mendelson 
column 4 lines 47-53, Figure 2). 

Mendelson does not specifically teach transfer of data values (channels) for a occuring only for changed 
data values. However, Shibata discloses a method whereby audio data is sent from the video server to the client 
station only during a "talk spurt", with constant frame rates occurring during periods of audio silence (i.e. a talk 
spurt changes the data values) (Shibata p.597, section 4.3 "Silence detected synchronization"). It would have 
been obvious to one of ordinary skill in the art at the time of the invention to apply Shibata to Mendelson's 
stream interleaving, providing Mendelson the benefit of file size and transmission reduction (streamlining) of 
the audio/video data streams. It is to be noted that in order to adjust frame rates as above, the server must know 
that a change in the data has occurred, which is typically achieved by comparing new data (the talk spurt) with 
the last recognized data value (previously recognized as empty - reflecting a period of silence). 

8. Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Mendelson, Durward, and 
Nielsen, as applied to claims 1, above, and further in view of Katseff et al. (hereinafter Katseff), U.S. 
Patent No. 5,822,537 issued October 1998. 
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In regard to dependent claim 5, Mendelson teaches a networking environment (Mendelson column 4 
lines 28-33). Mendelson does not specifically teach using the Internet for distribution. However, Katseff teaches 
an internet as an example of suggested types of networks that can be used (Katseff, column 3 lines 58-63). It 
would have been obvious to one of ordinary skill in the art at the time of the invention to apply Katseff to 
Mendelson, providing Mendelson the benefit of a widespread and familiar network. 

9. Claims 20, 30 are rejected under 35 U.S.C. 103(a) as being unpatentable over Mendelson, 
Durward, Nielsen, and Baumgartner. 

In regard to independent claims 20 and 30, claims 20 and 30 incorporate substantially similar subject 
matter as claimed in claims 1, 3, 10, 11, and are rejected along the same rationale. 

Response to Arguments 

10. Applicant presents no arguments at the present time. 

Conclusion 

11. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth 
in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the 
mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this 
final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory 
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period, then the shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no 
event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this 
final action. 

12. Any inquiry concerning this communication or earlier communications from the examiner should be 
directed to William L. Bashore whose telephone number is (571) 272-4088. The examiner can normally be 
reached on 1 1 :30am - 8:00pm EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Heather 
Herndon can be reached on (571) 272-4136. The fax phone number for the organization where this application 
or proceeding is assigned is 703-872-9306. 

13. Information regarding the status of an application may be obtained from the Patent Application 
Information Retrieval (PAIR) system. Status information for published applications may be obtained from 
either Private PAIR or Public PAIR. Status information for unpublished applications is available through 
Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you 
have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217- 
9197 (toll-free). 




WILLIAM BA£H0R£ 
PRIMARY EXAMINER 



April 27, 2007 



